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TODAY 

• SAT Validity 

• Previous SAT vs Current SAT 

• SAT Total, HSGPA, Writing 

• Differential Validity by Subgroup 

• Differential Prediction by Subgroup 

• Fairness and Subgroup Impact 
•ACT-SAT Concordance 
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Validity of the SAT 

University of California Validity Study 

• Examined the predictive validity of the new SAT for 33,356 students who 

• Completed the new SAT 

• Enrolled in a UC campus in the fall of 2006 

• http:/ /WWW. cair. org/conferences/CAIR2007/pres/Agronow.pdf 

• Results compared to previous UC study using the old SAT in 2004 

• Comparisons based on how well each measure predicted Freshman GPA 
at UC 


• SAT Critical Reading and Math slightly more predictive in 2006 than in 2004 

• SAT Writing slightly more predictive than the other SAT sections 

• SAT Writing (in 2006) slightly more predictive than Writing Subject Test had 
been (in 2004) 

• In 2004 study, High School GPA was slightly more predictive than SAT V+M 

• In 2006 study, SAT CR+M+W was slightly more predictive than High School 
GPA 
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Correlations between SAT and HSGPA with First Year College GPA at UC 
(see p, 4, 12, 15 of First Look) 



2004 

2006 

2006-2004 

HSGPA 

0.43 

0.44 

0.02 

SAT-CR 

0.38 

0.41 

0.03 

SAT-M 

0.30 

0.33 

0.03 

SAT-W 

0.40 

0.43 

0.03 

SAT-CR -H SAT-M 

0.39 

0.42 

0.03 

SAT-CR -r SAT-M + SAT-W 

0.41 

0.45 

0.03 

HSGPA -r SAT-CR + SAT-M 

0.50 

0.52 

0.02 

HSGPA -H SAT-CR -H SAT-M -H SAT-W 

0.51 

0.53 

0.02 


Note. In 2004, the SAT Subject Test in Writing was examined. 


* Multiple correlations computed from adj values 
“Correlations presented are not adjusted for restriction of range. 
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From Page 8 of Latest UC Validity Study... 


I’S ;VLUATION OF RESULTS 

lu re^.'iewms the result'- pre'iented in Fisuie& 1 - 4 the followiDg is ob&erin'ed: 

1) hi cne-to-oiie compari'scim, each predicator Tariible set iq 2006 accouDced for mjore 
variance in UC GPA :han id 2004. 

2) Relatione to the other variables in due models. High School GP.A accoimts for lesis 
variance in the predicdon of UC GPxA. in 2006 than it did in 2004. Mio of note, in 2004 
High School GPA (.1S2) compared v. ith the SAT I hfath - Verbal +SAT II U-nting 

{.1 .'2) accoimted for more variance in UC GPA. _n contra sc^ in 2006 the combination 
of SAT Critical Reading - Madi - ’Writing explained a h:t^e variaDce (.200) in 
UC GPA than High School GPA (.196). This difference tvas even larger for 
EnsineerinE students in 2006. 

i) SAT Critical Readme - Madi accoimted for more variance in most models in 2006 ±an 
in the SAT I Verbal + Math did in 2004. 

4) SAT Px.easoiiina Writine accoimted for more variance in most models in 2006 than in 
the SAT II Writine did in 2004 for all students, but not for Enaineerin? saidents. 

5) The combination of SAT Critical Reading - h-lath - Writing in 2006 contributed more 
than SAT Verbal + Math + S.AT II Writing did to :he prediction of UC GPA in 2004. 

6) For Engineering students, the SAT Math Subject test in 2006 accounted for more 
variance in UC GPA :han the S.AT II Ma:h test did in 2004. There was. however, little 
difference in this comparison for ALL students. 


^CollegeBoard 


connect to college success™ 
www.collegeboard.com 


Background on Recruitment for 
SAT Validity Study 

• Recruitment took place between 2005-2007 

• Data files were due to the Admitted Class Evaluation 
Service (ACES) system on 1 0/1 5/2007 

• Included 1 year performance data and retention to 2"'^ year on the 
entering class of Fall 2006 

• College Board aggregated files as they came in from 
ACES 

• Results of preliminary analyses (based on half of the 
sample) presented at College Board regional forums 
in Jan./Feb. 2008 

6 * Full rese arch report published in June 2008 
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College Board SAT Validity Study 


• 110 colleges participating in Validity Study (N = 
196,364) 

• Schools provided first year performance data for Fall 2006 
cohort through the Admitted Class Evaluation Service™ (ACES 
™) portal 

• Restrict sannple to students who completed the New 
SAT and submitted self reported High School GPA 
(N=151,316) 

• Additional analysis using actual HS GPA will be conducted. The 
institutional characteristics of the sample, as well as the total 
group, are presented on the next slide by key variables: 

• Region, Selectivity, Size, Control 
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Institutional Characteristics - DRAFT 


£ 


Variable 

P” 

Sample 

Population 

Region 

'mro 

1 1 5% 

1 1 6% 

MSRO 

24% 

18% 

NERO 

22% 

13% 

SRO 

11% 

25% 

SWRO 

11% 

10% 

WRO 

17% 

18% 

Selectivity 

under 50% 

24% 

20% 

50 to 75% 

54% 

44% 

over 75% 

23% 

36% 

Size 

Small: 750 to 1,999 undergrade 

20% 

18% 

Medium to Large: 2,000 to 7,499 
undergrade 

39% 

43% 

Large: 7,500 to 14,999 undergrade 

21% 

20% 

Very large: 15,000 or more undergrade 

20% 

19% 

' Control 

t 

Public 

43% 

57% 

Private 

57% 

43% 




SAT Validity Study — Descriptive 
Statistics (N=151,316) - DRAFT 


Predictor 

Mean 

SD 

National Mean 

HSGPA 

3.60 

0.50 

3.33 

SATCR 

560 

95.7 

503 

SATM 

579 

96.7 

518 

SATW 

554 

94.3 

497 

FGPA 

2.97 

0.71 

NA 
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SAT Validity Study — Correlations 
between Predictors - Raw (adjusted) 



10 


^CollegeBoard 


connect to college success™ 
www.collegeboard.com 


SAT Validity Study — Correlations for 
Predictors (N=151,316) - DRAFT 


Predictors 

Unadjusted 

R 

R* 

R*2 

HSGPA 

0.36 

0.54 

0.29 

SATW 

0.33 

0.51 

0.26 

SATCR 

0.29 

0.48 

0.23 

SATM 

0.26 

0.47 

0.22 

SAT CR+M 

0.32 

0.51 

0.26 

SAT CR+M+W 

0.35 

0.53 

0.26 

HSGPA + SAT 

0.46 

0.62 

0.38 
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So why do we adjust a correlation? 


Unrestricted Data 
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X 

Above - if a college 
admitted all students 
irrespective of SAT scores 
you would find a normal 
distribution of scores and 
FGPA and a high r 
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Below- the more selective the 
college, the less likely they 
admit students with low SAT 
scores - and they have far 
less students with low FGPA 
than in a population. 

irect Range Restriction on X 


r = .59 
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Unrestricted Data 


So why do we adjust a correlation? 

Below-We adjust a raw correlation 
to account for this restriction and to 
get us an estimate of the true 
validity of any measure. The same 
thing occurs anytime we restrict 
one variable in selection. 

irect Range Restriction on X 
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The Result (right) is that 
the entering class is 
restricted (to higher 
scoring students) which 
makes the correlation 
lower than it is in a 
representative population. 
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r= .70 
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SAT Validity Study — Incremental 
Validity (N=151,316) 


Predictors 

R* 

AR 

HSGPA (Add SAT CR + SAT M) 

.61 

.07 

HSGPA (Add SAT CR + SAT M + SAT W) 

.62 .08 

SAT CR + M (Add W) 

.53 

.02 


* Correlations corrected for restriction of range, pooled within-institution 
correlations 
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FGPA 


So what does a validity of .53 mean? 



600-890 900-1190 1200-1490 1500-1790 1800-2090 2100-2400 
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SAT Validity Study — Incremental Validity 

(N=151,316) (correlations adjusted for restriction of range and pooled within inst.) 



N 

SAT 

HSGPA 

SAT+HS 

L 



GPA 


CONTROL 

Private 

45,786 

.57 

.55 

.65 

Public 1 

105,530 

.52 

.53 

.61 

SELECTIVE 

Under 50% 

27,272 

.58 

.55 

.65 

50-75% 

84,433 

.53 

.54 

.62 

>75% 

39,611 

.51 

.54 

.60 
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What do past validity studies say? 

• DC - In 2006, the SAT accounted for more variance than HS 
grades, and more variance than the SAT M’s with writing or 
the former SAT V+M+SAT II W 

• Hezlett et. al., (2001) Meta-analysis of 3,000 validity studies - 
SAT predict grades in ail 4 yrs of college and cumulative GPA 

• SAT V .50, SAT M .48 HS GPA .61 , HS Rank .56 

• Bridgeman et. al., (2000) - SAT better than HSGPA in 
predicting college grades at most competitive colleges. 
Overall, SAT and HSGPA have greatest utility in more 
selective colleges. 

• Ramist et. al., (1993) - SAT better than HSGPA in predicting 
grades in many individual college courses 

(math, science, social science). 


SAT Validity Study — Future Directions 


• December 2008 

• SAT Validity Placement Report 

• How well does the SAT predict individual course grades in English 
composition, math, and other first-year courses? 

• 2009 

• Retention - How well does the SAT predict retention to 
the second year? 

• Differential Validity by Socio-Economic Status, Academic 
Rigor. 

• Impact of SAT above and beyond HSGPA in predicting 
college grades. 
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Summary of Results 

• SAT Writing has the highest correlation with FYGPA among 
the three individual SAT sections {Adj. r= 0.51). 

• As expected, the best combination of predictors of FYGPA 
is FISGPA and SAT scores (Adj. r=0.62), reinforcing the 
recommendation that colleges use both FISGPA and SAT 
scores to make the best predictions of student success. 

• The adjusted correlation of FISGPA and FYGPA is 0.54, 
which is only slightly higher than the multiple correlation of 
the SAT (Critical Reading, Math, and Writing combined) 
with FYGPA (Adj. r=0.53). 

• The SAT is slightly better in predicting FGPA than 
unweighted FIS grades at colleges admitting <50% of 
students, about even at colleges admitting 50%-75%, and 
less predicting at colleges admitting over 75% of students. 
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Summary of Results (continued) 


• The increment in predictive validity 
attributable to the SAT when HSGPA is 
taken into account is 0.08. 

• The increment in validity attributable to the 
Writing section over and above the Critical 
Reading and Math sections is 0.02. When 
HSGPA is also considered, the increment 
in validity attributable to the Writing section 
is 0.01. 
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Differential Validity and Prediction Report 


• Descriptive Statistics 

• Sample size by subgroups 

• Means by subgroups 

• Differential Validity 

• Correlation of SAT scores and HSGPA with FYGPA 
by Subgroups 

• Differential Prediction 


• Average Overprediction (-) and Underprediction (+) of 
FYGPA for SAT Scores and FISGPA by Subgroups 
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Sample size and mean of admission criteria by subgroups 


Subgroup 


n 

SAT-CR 

SAT-M 

SAT-W 

HSGPA 

Gender 

Male 

69,765 

564 

602 

550 

3.55 

Female 

81,551 

557 

559 

557 

3.65 

Race/ 

Ethnicity 

American Indian 

798 

544 

555 

529 

3.52 

Asian 

14,296 

562 

624 

562 

3.66 

African-American 

10,304 

506 

503 

498 

3.39 

Hispanic 

10,659 

524 

537 

520 

3.59 

No Response 

6,738 

587 

590 

576 

3.63 

Other 

4,497 

558 

572 

553 

3.57 

White 

104,024 

567 

584 

560 

3.62 

Best 

Language 

English 

140,559 

563 

579 

556 

3.60 

English and Another 

7,458 

531 

570 

534 

3.61 

Another Language 

1,718 

462 

605 

478 

3.61 

Not Stated 

1,581 

544 

559 

536 

3.53 

Totai 


151,316 

560 

579 

554 

3.60 


22 
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Correlation of SAT scores and HSGPA with FYGPA by Gender 


Subgroup 

Gender 

I 

Male 

Femaie 

k (institutions) 

107 

110 

N 

69,765 

81,551 

SAT-CR 

0.44 

0.52 

SAT-M 

0.45 

0.53 

SAT-W 

0.47 

0.54 

SAT 

0.50 


HSGPA 


0.54 

SAT, HSGPA 

0.59 

0.65 


Note- HSGPA and SAT are stronger predictors for females . Research on many 
tests consistently demonstrates grades and tests are slightly better in predicting 
^ female performance than male performance in college. 





























Correlation of SAT scores and HSGPA with FYGPA by Ethnicity 



Race/Ethnicity 

Subgroup 

American 

Indian 

Asian 

African- 

American 

Hispanic 

White 

k (inst.) 

16 

82 

83 

86 

109 

N 

384 

14,109 

10,096 

10,486 

104,017 

SAT-CR 

0.41 

0.41 

0.40 

0.43 

0.48 

SAT-M 

0.41 

0.43 

0.40 

0.41 

0.46 

SAT-W 

0.42 

0.44 

0.43 

0.46 

0.51 

SAT 

0.54 

0.48 

0.47 

0.50 

0.53 

HSGPA 

0.49 

0.47 

0.44 

0.46 

0.56 

SAT, HSGPA 

0.63 

0.56 

0.54 

0.57 

0.63 


Previous research has shown tests and grades are slightly less effective 
in predicting performance of African American students. 

























































Correlation of SAT scores and HSGPA with FYGPA by Best Language 


Subgroup Best Language 

English English & Another Another Language 
k 110 79 28 


n 

140,559 

7,237 

1,292 

SAT-CR 

0.49 

0.41 

0.28 

SAT-M 

0.47 

0.43 

0.34 

SAT-W 

0.52 

0.45 

0.32 

SAT 

0.54 



HSGPA 


0.42 

0.35 

SAT, HSGPA 

0.63 

0.55 

0.48 


SAT and HSGPA have less power in predicting FGPA of ELL students 
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Average Overprediction (-) and Underprediction (+) of FYGPA for 
SAT Scores and HSGPA by Gender 


Subgroup 

Gender 

Male 

Female 

k 

107 

110 

n 

69,765 

81,551 

SAT-CR 

-0.14 

0.12 

SAT-M 

-0.20 

0.17 

SAT-W 

-0.11 

0.10 

SAT 

-0.15 

0.13 

HSGPA 

-0.08 

0.07 

SAT, HSGPA 

-0.10 

0.09 


- means over predicting (Predicted FGPA for males is . 10 higher than actual 
26 GPA for males when SAT and HSGPA are used). + means under prediction 

C (Predicted FGPA for females that is .09 below actual FGPA. Consistent with 
past studies. 





























Average Overprediction (-) and Underprediction (+) ojFYGPA for 
SAT Scores and HSGPA by Ethnicity 



Race/Ethnicity 

Subgroup 

American 

Indian 

Asian 

African- 

American 

Hispanic 

White 

k 

103 

109 

108 

110 

110 

n 

798 

14,296 

10,304 

10,659 

104,024 

SAT-CR 

-0.26 

0.05 

-0.30 

-0.17 

0.04 

SAT-M 

-0.25 

-0.07 

-0.26 

-0.16 

0.05 

SAT-W 

-0.22 

0.04 

-0.26 

-0.16 

0.04 

SAT 

-0.22 

0.01 

-0.20 

-0.11 

0.03 

HSGPA 

-0.25 

0.02 

-0.32 

-0.27 

0.06 

SAT, HSGPA 

-0.20 

0.02 

-0.17 

-0.12 

0.03 


Also consistent with past research - The actual FGPA of under represented minorities average 
about .1 to .2 below predicted GPAs from SAT. HS grades consistent overpredict grades at a 
higher rate than tests. Over and underprediction are consistently reduced using both. 
























































Average Overprediction (-) and Underprediction (+) of 
FYGPA for SAT Scores and HSGPA by Best Language 



Best Language 

Subgroup 

English 

English & Another 

Another Language 

k 

110 

110 

102 

n 

140,559 

7,458 

1,718 

SAT-CR 

0.00 

-0.03 

0.40 

SAT-M 

0.01 

-0.09 

0.00 

SAT-W 

0.00 

-0.04 

0.37 

SAT 

0.00 

-0.02 

0.33 

HSGPA 

0.01 

-0.13 

0.06 

SAT, HSGPA 

0.00 

-0.03 

0.30 
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Summary of Results 


Similar to previous findings... 


• Differential Validity 

• SAT and HSGPA are more predictive of FYGPA for females, white 
students, and students who indicate English as their best language (larger 
correlations) 


• Differential Prediction 


• SAT and HSGPA tend to underpredict FYGPA for females; however, the 
magnitude is larger for the SAT 

• SAT and HSGPA tend to overpredict FYGPA for minority students; 
however, the magnitude is larger for HSGPA 

• SAT- CR & SAT- W tend to underpredict FYGPA for students whose best 
language is not English. SAT-M accurately predicts their FYGPA. 


SAT & HSGPA both tend to overpredict FYGPA for students whose best 
language is English and another language; however, the magnitude is larger 
for HSGPA. 
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These Results are no different than what has been reported for other 
standardized tests used in high schools, college and graduate admissions, as 
er data on group differences in college performance 
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Concordances: ACT and SAT 


• They are DONE! Expected to be posted on Collegeboard Ed 
professionals web site tomorrow!! 

http://professionals. collegeboard. com/data-reports-research 

• SAT Range to ACT Composite to SAT Single Score 

• Allow users to go from ACT to SAT or SAT to ACT 

• ACT Composite to SAT CR+W+M 

• ACT Composite to SAT CR+M 

• ACT Plus Writing to SAT W 

• Colleges wanting to compare SAT to ACT Plus Writing need to apply 
two concordance tables: 

• ACT Composite to SAT CR+M 

• ACT Writi ng to SAT Writing 
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We will also provide concordances between ACT Sum and SAT for 


c NCAA and other users (more precision). 
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Concordance: ACT and SAT 

• Sample: 

• 300,437 students who took both tests between 9/04 (for ACT) or 3/05 
(for SAT) and 6/06 - first High School graduating cohort since the 
revised tests were released. Students completed both tests. 

• Slightly more able SAT CR 523 (502), M 534 (51 8) W 51 8 (497). 

• Sample of students taking ACT Plus Writing was only 1 90,1 48 students 
and differed substantially in terms of ability and other factors. 

• If multiple scores, used closet to a Jr Spring Sr Fall Match. 

• A research report describing the sample, methodology and results 
will be jointly published by ACT, CB and ETS researchers this 
year. 

• Cautions - SAT has more score points than ACT - use of a range 
is most accurate. 
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Correlations: ACT Composite to SAT CR+M+W (.92) ACT W to 
SA T W (.85) 
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Questions? 
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Please go to our NEW REDESIGNED site for all 
of the DATA, RESEARCH and Answers to your 
data questions 

CB.com Education Professionals/Data-Report- 
Research 


http: //professionals, collegeboard. com/data-reports- 
research 


